EventGraD: Event-triggered communication in parallel machine learning

Communication in parallel systems imposes significant overhead which often turns out to be a bottleneck machine learning. To relieve some of this overhead, paper, we present EventGraD - an algorithm with event-triggered communication for stochastic gradient descent The main idea is modify the requirement at every iteration standard implementations learning communicating only when necessary certain iterations. We provide theoretical analysis convergence our proposed algorithm. also implement data-parallel training popular residual neural network used CIFAR-10 dataset and show that can reduce load by up 60% while retaining same level accuracy. In addition, combined other approaches such as Top-K sparsification decrease further maintaining